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PREFACE 


This report documents an independent research project undertaken by HQ 
AFMPC/DPMYOT to examine the effects of eliminating the 100 points, attributable 
to the Specialty Knowledge Test (SKT), from the Weighted Airman Promotion 
System (WAPS) on the rank-ordering of members in SKT-exempt AFSCs. This 
project was reviewed by AL/HRM and was supported by the computer 
programming staff of AL/HR. A special thanks goes to Ms Doris Black and her 
staff for ensuring data availability and statistical analyses support 








AN APPROACH FOR EQUALIZING TEST SCORES FOR SKT-EXEMPT AFSCs 


Introduction 

Background 

Hie Weighted Airman Promotion System (WAPS) is used by the US Air Force to 
promoted enlisted personnel to the grades of E-5 through E-7. WAPS is composed 
of six weighted factors which combine measures of professional knowledge, job 
performance, and experience (longevity): Specialty Knowledge Test (SKT), 
Promotion Fitness Exam (PFE), Enlisted Performance P.eports (EPR), Time in 
Grade (TIG), Time in Service (TIS). and Decu'aiioiif (DEC). Of a total of 460 
possible points, the SKT and the PFE account for 100 points each; EPRs contribute 
135 possible points; TIS, 40 points; TIG, 60 points; and DEC, 25 points. An 
airman's score on each factor is totaled and the sum of the total WAPS points is 
rank-ordered with all other members within each Air Force Specialty Code (AFSC) 
at each grade. The promotion percentage allocated by HQ Air Force for a specific 
grade is then used to select those who have the highest total WAPS points for 
promotion in that promotion cycle. While this approach may seem complicated, it is 
the most objectiv? anJ open approach among the Services and has been accepted by 
enlisted personnel since its inception in 1970. 

WAPS was developed in 1968 and fully implemented in 1970 to provide an enlisted 
promotion system that was objective and fair for all considered for promotion. As 
an alternative to what was then perceived as a biased and secretive approach to 
promotion, WAPS was a system fully under the control of each enlisted member. 
After implementation, WAPS has undergone revalidation in 1972,1977, and 1986. 


1 









Using a policy-capturing approach for each revalidation, the data indicated that 
different promotion formulae should be applied to those in different grades (Treat, 
et al, 1987). However, it was determined that different weighting schemes would 
lead to a less understandable, and potentially less acceptable, system. The less 
empirical, and more policy driven, original weights have been used since WAPS 
development 

Almost all Air Force Specialty Codes (AFSCs) use all six components of WAPS. 
However, a few AFSCs such as recruiters and couriers, do not have a SKT 
developed for them. These special AFSCs take members from their primary AFSC 
to perform varied jobs in these diverse AFSCs, prohibiting the development of an 
AFSC-wide SKT. Other AFSCs have very small number of members making it not 
cost-effective to develop a SKT. Therefore, members in these SKT-exempt AFSCs 
must compete for promotion with a maximum of360 point rather than the standard 
460 points. This results in a disproportionate weighting given to the longevity 
factors, TIG and TIS (28% versus 22%). This statistical anomaly reduces the 
chances for promotion of those members more junior (less TIG and US) as 
compared to those members competing with both SKT and PFE. Even though all 
members of a SKT-exempt AFSC compete within that AFSC only, a perception of 
inequity with the rest of the enlisted force may exist, given the increased effects of 
longevity on the promotion score. 

Purpose 

This paper examines whether double-weighting the PFE portion of the WAPS 
formula, as well as other possible approaches, could reduce the effects of Time in 
Grade (TIG) and Time in Service (TIS) for SKT-exempt AFSCs. In their 
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revalidation of WAPS, Treat, et al (1987), examined the possibility of establishing "a 
separate weighting formula for AFSCs which have no SKT for any given year" (p. 
4). In the policy capturing equations which resulted from raters' review of 100 
records, the PFE had weights from three to eight times larger than when the SKT 
was included. They concluded the SKT-exempt AFSCs "should have a separate 
formula with a larger PFE weight" to compensate for the loss of the substantial 
variability contributed to the WAPS equation by the SKT. 

Method 

Subjects 

WAPS component scores were taken from the records of 6643 E-4s competing for 
promotion to E-5,7374 E-5s competing for promotion to E-6, and 5405 E-6s 
competing for promotion to E-7 in sixteen SKT-exempt AFSCs during FY 88 and 
FY 89. These AFSCs were randomly selected from each major occupational group 
(i.e^ maintenance, electronics, operations, etc.). For comparison, the records of 
65,285 E-4s, 54,308 E-5s, and 28,877 E-6s were examined. These members came 
from 23 randomly selected AFSCs across all major occupational groups which took 
both the SKT and the PFE. The data for the non-exempt AFSCs was taken from the 
same promotion cycles as the SKT-exempt AFSCs. 

Instruments 

The Promotion Fitness Examination (PFE) and the Specialty Knowledge Test (SKT) 
are norm-referenced instruments used to assess general professional knowledge and 
AFSC-spedfic (not job-specific) knowledge possessed by enlisted members 
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competing for promotion within each AFSC. Both tests are revised annually by 
senior enlisted subject-matter experts and personnel trained in test development. 
The PFE is taken by all enlisted personnel eligible for promotion to E-5 .‘hrough E-7 
while the SKT is taken only by those in AFSCs designated by the Air Force Military 
Personnel Center. Each instrument contains 100,4-alternative multiple choice 
items. There are no parallel forms of these instruments. Typical means range from 
45 to 65, with the majority ranging from 50 to 55. Items are selected based on 
statistical sufficiency, content validity, and referencibility to specific study materials 
published 6 months prior to the testing cycle. Reliability, as assessed by 
Cbronbach's Alpha range from .85 to .98. 

Analyses 

To determine the effects of double-weighting the PFE on promotion selections, the 
PFE scores of individuals in the 16 SKT-exempt AFSCs were doubled, added to 
their respective total WAPS scores, and then all members were rank-ordered based 
on their double weighted WAPS score. The same promotion percentage was then 
applied to the double weighted rank order. In order to determine eqnivalency of 
performance on the PFE between the 23 non-exempt AFSCs, means and standard 
deviations were computed and compared. A Pearson product moment correlation 
matrix for the combined 23 non-exempt AFSCs was calculated for all WAPS 
components. This analysis was performed for each grade considered and each 
promotion cycle (respective year of promotion testing). To examine whether 
members who were selected using the current promotion formula for SKT-exempt 
AFSCs were either selected or non selected when the PFE score was doubled, a Chi- 
square analysis was conducted for each grade. A Phi coefficient was also calculated 
for each Chi-square. 
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Results 


The means and standard deviations within each grade between promotion cycles 
were compared and revealed some significant differences on WAPS components. 
For those E-5s (cycle 89A5) promoted under the current system (hereafter called 
single-weighted (SW» and not under the experimental system (hereafter called 
double-weighted (DW)), their average Time in Grade (TIG) was 4 years, 8 months 
(TIG points are awarded as 1/2 point per month after pin-on of the current grade). 
However, those E-5s promoted after double-weighting only had 3 years 4 months 
TIG (see Figure 1). This pattern remained consistent for the other grades as well. 
Figure 2 shows the differences in Time in Service (TIS), revealing that TIS effects 
are not as prominent for more junior enlisted personnel (E-5s) as they are when 
seeking promotion to E-6 and E-7. It remains, however, that if more weight were 
placed on the knowledge portion of the WAPS formula, less senior personnel would 
be promoted. Enlisted Performance Reports (EPRs) contributes very little 
variability to the WAPS formula due to a strong ceiling effect. As grade increases, 
EPR scores are practically the same for members competing for higher rank. 
However, Figure 3 shows a small but definite difference between single-weighted 
and double-weighted groups, with the single-weighted group performing more 
poorly than those in the double-weighted group. 

The Chi-square analyses, shown in Table 1, depict a significant change in those who 
would have been promoted if the PFE had been double-weighted. For those 
competing for promotion to E-5 (AS), 205 individuals would have been promoted by 
using a double-weighted PFE that were not promoted under the current promotion 
system. This 31 percent shift was highly significant (%2= 2912, p< .0001). 
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Figure 1. Time in Grade Means for Three Promotion Cycles. 
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Table 1 

CHI-SQUARE ANALYSIS OF DOUBLE-WEIGHTING EFFECTS 


Non-select 
DOUBLE WEIGHT 
Select 


CYCLE 89A5 
SINGLE WEIGHT 
Nofvselecl Select 


5767 

205 

205 

456 


gXP.tf.WA8 

SINGLE WEIGHT 
Noiteeiect Select 


Noftedect 
DOUBLE WEIGHTj 
Select 


6247 

249 

249 

629 


CYCLE 89A7 
SINGLE WEIGHT 
Norhsetect Select 


2550 

171 

171 

513 


For those competing for promotion to E-6 (A6), the 28 percent shift from single- 
weighted to double-weighted was also significant (y2= 3391, p< .0001). For those 
competing for promotion to E-7 (AT), the shift of 171 personnel, while less dramatic, 
was sill significant (^2= 376, p< .0001). This reduced effect may be the result of 
more homogeneity among competing personnel. 

Table 2 shows the intercorrelations among all WAPS components for those 
competing for promotion to E-5 in the 23 non-exempt AFSCs. The strong 
correlation between SKT and PFE (r= .42, p<.001) demonstrates the 
interrelationship of two measures of Air Force and specialty knowledge. The 
relationship between TIG and ITS (r= .886, pcOOl) is also logical, since both are 
measures of longevity. The moderate relationship between decorations and TIG 
and TIS is strongest with personnel competing for promotion to E-5 (r= .23 and r= 
32, pc .001, respectively) as compared to E-6 and E-7 promotion cycles. 
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Table 2 

AVERAGE CORRELATIONS AMONG WAPS FACTORS 
FOR CYCLES 89A5 AND 90A5 IN 23 NON-EXEMPT AFSCs 



NOTE: AU correlations significant at p < .01 


Table 3 contains correlations among WAPS components for those competing for 
promotion to E-6 in the non-exempt AFSCs. The relationship between SKT and 
PFE is stronger in this group than for the E-5 group (n= .55, pc.001). The same 
strong relationship exists between TIG and T1S. However, it appears that the 


Table 3 

AVERAGE CORRELATIONS AMONG WAPS FACTORS 
FOR CYCLES 89A6 AND 90A6 IN 23 NON-EXEMPT AFSCs 
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longer an individual is in the Air Force (TIS), the less well that individual will do on 
the SKT and PFE (r= -.27 and r= -26, p< .001, respectively). Also, the effect of TIG 
and TIS on decoration points has moderated, probably due to a restriction in range 
effect 


Table 4 

AVERAGE CORRELATIONS AMONG WAPS FACTORS 
FOR CYCLES 89A7 AND 90A7 IN 23 NON-EXEMPT AFSCs 

SKT PFE TIG TIS EPR DEC 

SKT 
PFE 
TIG 
TIS 
EPR 
DEC 

NOTE: All correlations significant at p •: .01 

Table 4 illustrates the correlations among WAPS components for those competing 
for promotion to E-7 in the non-exempt AFSCs. Again a strong relationship exists 
the SKT and the PFE (r= .53, pc .001). Also, the correlation between the SKT and 
PFE with TIS is even stronger in the negative direction (r= -.44 and r= -.45, p< .001, 
respectively). TIG and TIS are still strongly related, but less so than for the E-5 and 
E-6 groups (r= .71, p< .001). The relationship between TIG and ITS with 
decorations has all but disappeared probably due to the fact that as senior NCOs, 
most have nearly the same decoration points. 
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Discussion 


The data show two different and interesting profiles for those promoted under the 
single-weighted and double-weighted systems. By double-weighting the PFE, a 
quite different group of personnel were selected for promotion. Those selected 
under the double weighted system were more junior in grade and performed 
significantly better on the PFE than did the single-weighted group. Since EPRs have 
little variability, the points awarded for performance did not play a significant role 
in separating the two groups. 

When looking at the correlations among the WAPS components, some expected and 
some not so expected results are revealed. Those more senior in SKT-exempt 
AFSCs seem to be more effected by the increased reliance on TIG and ITS. 
Typically, those that perform well on the PFE but are more junior have less of a 
chance at being promoted. The issue is what type of individual does the system 
want or need to promote. The choices are those that test well and perform well or 
those that don't test as well, perform adequately, and have more seniority. 

There are two approaches to altering the WAPS system to eliminate the differences 
in promotion formulae for SKT-exempt and non-exempt AFSCs. Doubling the PFE 
score is a simple linear change that can be easily explained and understood by the 
enlisted force. However, there are potential problems with this approach. It 
assumes that the PFE is statistically equivalent to the SKT which, based on 
correlational data, it is not. The SKT measures something different than the PFE 
(only 16 percent shared variance), thereby creating a possible content validity issue. 
It is, however, the easiest adaptation to the current system. 
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The second, more statistically pure, approach is to obtain the correlation by grade 
between the SKT and PFE for the non-exempt AFSCs and create an adjusted 
estimate of the SKT for the exempt AFSCs. The formula needed to make this 
estimate is shown below: 

Double-weighted SKT = (p * (ffy / a x )) * (X - X) + Y 

The major problem with this solution is that it appears to the enlisted force as some 
magical numbers manipulation and cannot be directly related to the original 
formula. A difficult and timely education program would need to be provided to 
gain enlisted acceptance of the double-weighted formula, a central focus of the 
current system. Further work is expected in this area to ensure a fair and equitable 
promotion system for all Air Force enlisted personnel. 
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SUMMARY 


The results of this study indicate that double-weighting the Promotion 
Fitness Examination (PFE) portion of the WAPS formula for SKT-exempt AFSCs 
has a significant effect on those selected for promotion. By using the double- 
weighted PFE score for SKT-exempt AFSCs, the effects of longevity are made 
relatively equivalent to those AFSCs using both SKT and PFE in the WAPS 
formula. This approach seems easier to understand by the enlisted force than 
applying a statistical adjustment. 

This approach is being briefed and staffed at HQ AFMPC for eventual 
disposition. Whatever the results of the review may be, the purpose of this 
investigation was to provide the WAPS with an easily explainable and wholly 
defensible adjustment to total WAPS points for SKT-exempt AFSCs. 
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